HoloLens 2 Sensor Streaming

We present a HoloLens 2 server application for streaming device data via TCP in real time. The server can stream data from the four grayscale cameras, depth sensor, IMU, front RGB camera, microphone, head tracking, eye tracking, and hand tracking. Each sent data frame has a timestamp and, optionally, the instantaneous pose of the device in 3D space. The server allows downloading device calibration data, such as camera intrinsics, and can be integrated into Unity projects as a plugin, with support for basic upstream capabilities. To achieve real time video streaming at full frame rate, we leverage the video encoding capabilities of the HoloLens 2. Finally, we present a Python library for receiving and decoding the data, which includes utilities that facilitate passing the data to other libraries. The source code, Python demos, and precompiled binaries are available at https://github.com/jdibenes/hl2ss.Comment: Technical repor

DDM-NET: End-to-end learning of keypoint feature Detection, Description and Matching for 3D localization

In this paper, we propose an end-to-end framework that jointly learns keypoint detection, descriptor representation and cross-frame matching for the task of image-based 3D localization. Prior art has tackled each of these components individually, purportedly aiming to alleviate difficulties in effectively train a holistic network. We design a self-supervised image warping correspondence loss for both feature detection and matching, a weakly-supervised epipolar constraints loss on relative camera pose learning, and a directional matching scheme that detects key-point features in a source image and performs coarse-to-fine correspondence search on the target image. We leverage this framework to enforce cycle consistency in our matching module. In addition, we propose a new loss to robustly handle both definite inlier/outlier matches and less-certain matches. The integration of these learning mechanisms enables end-to-end training of a single network performing all three localization components. Bench-marking our approach on public data-sets, exemplifies how such an end-to-end framework is able to yield more accurate localization that out-performs both traditional methods as well as state-of-the-art weakly supervised methods

NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization

Monocular 3D object localization in driving scenes is a crucial task, but challenging due to its ill-posed nature. Estimating 3D coordinates for each pixel on the object surface holds great potential as it provides dense 2D-3D geometric constraints for the underlying PnP problem. However, high-quality ground truth supervision is not available in driving scenes due to sparsity and various artifacts of Lidar data, as well as the practical infeasibility of collecting per-instance CAD models. In this work, we present NeurOCS, a framework that uses instance masks and 3D boxes as input to learn 3D object shapes by means of differentiable rendering, which further serves as supervision for learning dense object coordinates. Our approach rests on insights in learning a category-level shape prior directly from real driving scenes, while properly handling single-view ambiguities. Furthermore, we study and make critical design choices to learn object coordinates more effectively from an object-centric view. Altogether, our framework leads to new state-of-the-art in monocular 3D localization that ranks 1st on the KITTI-Object benchmark among published monocular methods.Comment: Paper was accepted to CVPR 202

Commercialization of an Ergonomic Scalpel

The goal of this project was to analyze the feasibility of commercializing a newly patented ergonomic surgical scalpel. In order to achieve this, we collected data, performed a feasibility analysis, and provided recommendations. We evaluated four areas: the medical device market, manufacturing, consumer research, and intellectual property. We concluded that this product would not be viable for commercialization as a stand-alone product. Alternative methods to commercialization may be required in order to generate future success if this product reaches market

Efficient video collection association using geometry-aware Bag-of-Iconics representations

Abstract Recent years have witnessed the dramatic evolution in visual data volume and processing capabilities. For example, technical advances have enabled 3D modeling from large-scale crowdsourced photo collections. Compared to static image datasets, exploration and exploitation of Internet video collections are still largely unsolved. To address this challenge, we first propose to represent video contents using a histogram representation of iconic imagery attained from relevant visual datasets. We then develop a data-driven framework for a fully unsupervised extraction of such representations. Our novel Bag-of-Iconics (BoI) representation efficiently analyzes individual videos within a large-scale video collection. We demonstrate our proposed BoI representation with two novel applications: (1) finding video sequences connecting adjacent landmarks and aligning reconstructed 3D models and (2) retrieving geometrically relevant clips from video collections. Results on crowdsourced datasets illustrate the efficiency and effectiveness of our proposed Bag-of-Iconics representation

Warped K-Means: An algorithm to cluster sequentially-distributed data

[EN] Many devices generate large amounts of data that follow some sort of sequentiality, e.g., motion sensors, e-pens, eye trackers, etc. and often these data need to be compressed for classification, storage, and/or retrieval tasks. Traditional clustering algorithms can be used for this purpose, but unfortunately they do not cope with the sequential information implicitly embedded in such data. Thus, we revisit the well-known K-means algorithm and provide a general method to properly cluster sequentially-distributed data. We present Warped K-Means (WKM), a multi-purpose partitional clustering procedure that minimizes the sum of squared error criterion, while imposing a hard sequentiality constraint in the classification step. We illustrate the properties of WKM in three applications, one being the segmentation and classification of human activity. WKM outperformed five state-of- the-art clustering techniques to simplify data trajectories, achieving a recognition accuracy of near 97%, which is an improvement of around 66% over their peers. Moreover, such an improvement came with a reduction in the computational cost of more than one order of magnitude.This work has been partially supported by Casmacat (FP7-ICT-2011-7, Project 287576), tranScriptorium (FP7-ICT-2011-9, Project 600707), STraDA (MINECO, TIN2012-37475-0O2-01), and ALMPR (GVA, Prometeo/20091014) projects.Leiva Torres, LA.; Vidal, E. (2013). Warped K-Means: An algorithm to cluster sequentially-distributed data. Information Sciences. 237:196-210. https://doi.org/10.1016/j.ins.2013.02.042S19621023

The InfraRed Imaging Spectrograph (IRIS) for TMT: latest science cases and simulations

The Thirty Meter Telescope (TMT) first light instrument IRIS (Infrared Imaging Spectrograph) will complete its preliminary design phase in 2016. The IRIS instrument design includes a near-infrared (0.85 - 2.4 micron) integral field spectrograph (IFS) and imager that are able to conduct simultaneous diffraction-limited observations behind the advanced adaptive optics system NFIRAOS. The IRIS science cases have continued to be developed and new science studies have been investigated to aid in technical performance and design requirements. In this development phase, the IRIS science team has paid particular attention to the selection of filters, gratings, sensitivities of the entire system, and science cases that will benefit from the parallel mode of the IFS and imaging camera. We present new science cases for IRIS using the latest end-to-end data simulator on the following topics: Solar System bodies, the Galactic center, active galactic nuclei (AGN), and distant gravitationally-lensed galaxies. We then briefly discuss the necessity of an advanced data management system and data reduction pipeline.Comment: 15 pages, 7 figures, SPIE (2016) 9909-0

Solving the conundrum of intra-specific variation in metabolic rate: A multidisciplinary conceptual and methodological toolkit

Researchers from diverse disciplines, including organismal and cellular physiology, sports science, human nutrition, evolution and ecology, have sought to understand the causes and consequences of the surprising variation in metabolic rate found among and within individual animals of the same species. Research in this area has been hampered by differences in approach, terminology and methodology, and the context in which measurements are made. Recent advances provide important opportunities to identify and address the key questions in the field. By bringing together researchers from different areas of biology and biomedicine, we describe and evaluate these developments and the insights they could yield, highlighting the need for more standardisation across disciplines. We conclude with a list of important questions that can now be addressed by developing a common conceptual and methodological toolkit for studies on metabolic variation in animals